AD 


Integrated  Stereo  Infrared  and  Color  Human  Detection 
on  the  ART  platform 

Final  Report 
by 

M.  Bertozzi,  A.  Broggi,  M.  Felisa,  S.  Ghidoni 
December  2006  -  March  2007 

United  States  Army 

EUROPEAN  RESEARCH  OFFICE  OF  THE  U.S.  ARMY 
London,  England 

CONTRACT  NUMBER  N62558-07-P-0029 

UNIVERSITA  DI  PARMA 
Parma,  Italy 

Approved  for  Public  Release;  distribution  unlimited 

The  Research  reported  in  this  document  has  been  made  possible  through  the  support  and  sponsorship  of  the  U.S. 
Government  through  its  European  Research  Office  of  the  U.S.  Army. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


Public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data 
sources,  gathering  and  maintaining  the  data  needed,  and  completing  and  reviewing  this  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other 
aspect  of  this  collection  of  information,  including  suggestions  for  reducing  this  burden  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports, 
1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA  22202-4302,  and  to  the  Office  of  Management  and  Budget,  Paperwork  Reduction  Project  (0704-0188),  Washington,  DC 
20503 


1.  AGENCY  USE  ONLY  (Leave 
blank) 

2.  REPORT  DATE 

April  1,  2006 

3.  REPORT  TYPE  AND  DATES  COVERED 

Final  report  Dec  2006-Mar  2007 

4.  TITLE  AND  SUBTITLE 

Integrated  Stereo  Infrared  and  Color  Human  Detection  on  the 
ART  platform 

5.  FUNDING  NUMBERS 

N62558-07-P-0029 

6.  AUTHOR(S) 

M.  M.  Bertozzi,  A.  Broggi,  S.  Ghidoni,  M. 

Felisa 

7.  PERFORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Dip.  Ing.  Informazione 

Universita  di  Parma 

Parco  area  delle  Scienze,  181A 

1-43100  Parma 

ITALY 

8.  PERFORMING  ORGANIZATION 

REPORT  NUMBER 

DII-VISLAB-TR005-2006 

9.  SPONSORING  /  MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

US  Naval  Regional  Contracting  Center,  Det.  London 

Government  Buildings,  Block  2,  Wing  12 

Lime  Grove,  Ruislip,  Middlesex  HA4  8BX, 

UNITED  KINGDOM 

10.  SPONSORING  /  MONITORING 

AGENCY  REPORT  NUMBER 

11.  SUPPLEMENTARY  NOTES 


12a.  DISTRIBUTION  /  AVAILABILITY  STATEMENT 


12b.  DISTRIBUTION  CODE 


13.  ABSTRACT  (Maximum  200  Words) 


This  report  presents  the  research  activities  within  the  framework  of  N62558-05-P-0380 
contract  for  the  development  of  a  Human  Shape  localization  system  by  means  of  a  4  cameras 
vision  system  consisting  of  2  daylight  and  2  far  infrared  cameras. 


The  main  idea  is  to  exploit  the  advantages  of  both  far  infrared  and  visible  cameras  to 
develop  a  system  that  combines  the  advantages  of  using  far  infrared  or  daylight 
technologies . 

In  particular,  this  report  details  a  system  for  detecting  pedestrians  using  histogram  of 
oriented  gradients  by  means  of  support  vector  machines. 


14.  SUBJECT  TERMS 

Pedestrian  detection,  oriented  gradients,  svm,  far  infrared 

15.  NUMBER  OF  PAGES 

9 

16.  PRICE  CODE 

17.  SECURITY 
CLASSIFICATION 

OF  REPORT 

18.  SECURITY 
CLASSIFICATION 

OF  THIS  PAGE 

19.  SECURITY  CLASSIFICATION 
OF  ABSTRACT 

20.  LIMITATION  OF 
ABSTRACT 

NSN  7540-01-280-5500 


Standard  Form  298  (Rev.  2-89) 

Prescribed  by  ANSI  Std.  Z39-18 
298-102 


Abstract 


This  report  presents  the  research  work  developed  under  contract  number  N62558-07-P-0029  with  the 
aim  of  localizing  human  shapes  using  two  stereo  systems  consisting  of  two  daylight  and  two  far-infrared 
(FIR)  cameras. 


Contents 

1  Introduction  2 

2  Related  work  2 

3  System  description  3 

4  Bounding  boxes  analysis  4 

4.1  HoG  descriptor .  4 

4.2  SVM  classifier .  5 

5  Results  6 

6  Conclusion  7 

A  List  of  publications  produced  within  the  whole  project  9 

B  List  of  scientific  personnel  9 


1 


1  Introduction 


This  report  details  the  subsystem  developed  for  extracting  the  object  shape,  in  order  to  provide  a  vote  based 
on  it.  However,  the  evaluation  of  the  object  shape  is  currently  under  development,  therefore  only  results  on 
shape  extraction  are  given. 

When  a  shape  is  extracted,  a  function  has  been  implemented  to  detect  the  presence  of  a  pedestrian. 
To  analyze  the  shape,  it  extracts  the  corresponding  image  and  describe  it  thanks  to  histograms  of  oriented 
gradient.  This  method  has  been  introduced  by  [11]  and  aims  at  splitting  the  image  into  small  regions  and 
computing  histograms  of  oriented  gradients  for  each  region  and  merging  all  histograms  to  describe  the  image. 
The  idea  consists  in  describing  the  shape  of  the  object  thanks  to  the  information  of  gradient  orientation. 
This  characterization  is  finally  used  with  a  linear  Support  Vector  Machine  classifier  [30] ,  which  is  the  state  of 
the  art  within  the  classification  methods  and  has  proved  to  be  very  efficient  in  such  case  of  high  dimensional 
data. 

The  report  is  organized  as  follows:  section  2  describes  related  work  in  pedestrian  detection  systems  based 
on  artificial  vision.  The  pedestrian  detection  system  is  discussed  in  3  and  the  submodule  for  the  analysis  of 
the  areas  of  attention  is  detailed  in  section  4.  Results  are  discussed  in  paragraph  5  and,  finally,  section  6 
ends  the  report  with  some  final  remarks. 


Visible 


Detection 


Symmetry  Filters 


Figure  1:  Overall  algorithm  flow.  This  report  describes  the  Histogram  of  Oriented  Gradients  (HOG)  and 
Support  Vector  Machines  (SVM)  stages. 


2  Related  work 

For  the  U.S.  Army  the  use  of  vision  as  a  primary  sensor  for  the  detection  of  human  shapes  is  a  natural  choice 
since  cameras  are  non-evasive  sensors  and  therefore  do  not  emit  signals. 

Vision-based  systems  for  pedestrian  detection  have  been  developed  exploiting  different  approaches;  like 
the  use  of  monocular  [23,  17]  or  stereo  [24,  27]  vision.  Daylight  cameras  are  widely  used  for  pedestrian 
detection,  but,  in  the  last  years,  thanks  to  the  decreasing  costs  and  to  the  improved  technology,  also  infra 
red  devices  have  been  employed  for  vision  based  pedestrian  detection  [12,  2]. 

Many  systems  based  on  the  use  of  a  stationary  camera  employ  simple  segmentation  techniques  to  obtain 
foreground  region;  but  this  approach  fails  when  the  pedestrians  has  to  be  detected  from  moving  platforms. 
Most  of  the  current  approaches  for  pedestrian  detection  using  moving  cameras  treats  the  problem  as  a 
recognition  task:  a  foreground  detection  is  followed  by  a  recognition  step  to  verify  the  presence  of  a  pedestrian. 
Some  systems  use  motion  detection  [10]  or  stereo  analysis  [27,  25]  as  a  means  of  segmentation. 

Other  systems,  substitute  the  segmentation  step  with  a  focus-of-attention  approach,  where  salient  regions 
in  feature  maps  are  considered  as  candidates  for  pedestrians,  recognition  approach  is  used  to  validate  the 
presence  of  pedestrians.  In  the  GOLD  system  [6]  a  DSI-based  approach  is  used  to  detect  potential  pedestrians 
and  the  recognition  is  obtained  by  a  number  of  independent  validators.  A  similar  approach  is  used  in  [14] 
where  region  of  interests  are  preliminary  selected  using  a  stereo  vision  approach  and  following  shape-based 
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and  texture-based  stages  are  used  for  the  validation.  In  [9]  the  local  image  entropy  directs  the  focus-of- 
attention  and  is  followed  by  a  model-matching  module. 

The  most  critical  problem  is  to  quickly  and  reliably  determine  whether  a  candidate  region  contains  a 
pedestrian.  The  computer  vision  approaches  dealing  with  the  detection  and  recognition  of  pedestrians  can 
be  divided  into  three  categories:  model-based  methods  that  use  models  set  of  the  human  body,  holistic 
methods  which  model  the  appearance  or  shape  of  the  entire  body,  and  motion-based  recognition  methods 
which  use  a  model  of  the  movements  of  the  whole  body  or  individual  parts  of  it.  For  examples,  [10,  9]  use  the 
periodicity  of  human  gait  or  gait  patterns  for  pedestrian  detection.  The  model-based  approaches  seems  to 
be  more  reliable  than  shape  based  ones,  but  they  require  temporal  information  and  are  unable  to  correctly 
classify  pedestrians  that  are  still  or  have  an  unusual  gait  pattern. 

Conversely,  shape-based  approaches  exploit  the  pedestrians’  appearance,  so  can  detect  both  moving  and 
stationary  people  [17,  21].  In  model-based  approach,  the  challenge  is  to  model  the  several  variations  in  the 
shapes,  pose,  size  and  appearance  of  humans  and  their  backgrounds.  Basic  shape  analysis  methods  consist  in 
matching  a  template  onto  candidate  foreground  regions.  In  [13]  a  tree-based  hierarchy  of  human  silhouettes 
is  constructed  and  the  matching  follows  a  coarse-to-fine  approach.  In  [20,  26]  probabilistic  templates  are 
used  to  take  in  account  the  possible  variations  in  human  shape. 

The  method  shown  in  this  report  falls  in  the  category  of  holistic  approaches.  The  advantage  of  this  choice 
is  its  general  applicability;  it  learns  the  shape  model  from  examples,  and  hence  is  not  specific  to  humans 
and  can  be  readily  applied  to  the  recognition  of  other  classes  of  objects.  Learning  based  classifiers,  such  as 
Neural  Network  [28,  29,  19]  and  Support  Vector  Machine  (SVM)  [32,  1,  7,  31,  18,  30],  have  mostly  been  used 
for  the  validation.  As  an  example,  [16]  uses  a  neural  net  to  discriminate  pedestrians  based  on  the  disparity 
map  obtained  from  a  stereo  pair.  In  [15]  a  vertical  and  horizontal  3x3  Sobel  edge  detector  is  used  to  extract 
pedestrian  shape  features  and  trained  two  SVM-basecl  classifiers  for  detecting  pedestrians  in  front/rear  pose 
respectively.  In  [32]  SVM-based  classifier  is  trained  using  grayscale  features  in  a  infrared  image. 

3  System  description 

The  active  contours-based  algorithm  described  in  this  work  has  been  developed  as  a  part  of  a  tetra-vision 
based  pedestrian  system  [3,  6].  The  whole  architecture  is  based  on  the  simultaneous  use  of  two  far  infrared 
and  two  daylight  cameras.  Thanks  to  this  approach,  the  system  is  able  to  detect  obstacles  and  pedestrians 
when  the  use  of  infrared  devices  is  more  appropriate  (night,  low  illumination  conditions. . . )  or,  conversely, 
in  the  case  visible  cameras  are  more  suitable  for  the  detection  (hot,  sunny  environments. . . ). 


Figure  2:  Examples  of  typical  scenarios  in  FIR  and  visible  images. 
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In  fact,  FIR  images  convey  a  type  of  information  that  is  very  different  from  those  in  the  visible  spectrum. 
In  the  infrared  domain  the  image  of  an  object  depends  on  the  amount  of  heat  it  emits,  namely,  it  is  generally 
related  to  its  temperature  (see  fig.  2) .  Conversely,  in  the  visible  domain,  objects  appearance  depends  on  how 
the  surface  of  the  object  reflects  the  incident  light  as  well  as  on  the  illumination  conditions. 

Since  humans  usually  emit  more  heat  than  other  objects  like  trees,  background,  or  road  artifacts,  the 
thermal  shape  can  be  often  successfully  exploited  for  pedestrian  detection.  In  such  cases,  pedestrians  are 
in  fact  brighter  than  the  background.  Unfortunately,  other  road  participants  or  artifacts  emit  heat  as  well 
(cars,  heated  buildings. . . ).  Moreover,  infrared  images  are  blurred,  have  a  poor  resolution  and  the  contrast 
is  low  compared  with  rich  and  colorful  visible  images. 

Consequently,  both  visible  and  far  infrared  images  are  used  for  reducing  the  search  space. 

Figure  1  depicts  the  overall  algorithm  flow  for  the  complete  pedestrian  system.  Different  approaches 
have  been  developed  for  the  initial  detection  in  the  two  image  domains:  warm  areas  detection,  vertical  edges 
detection,  and  an  approach  based  on  the  simultaneous  computation  of  disparity  space  images  in  the  two 
domains  [3,  6]. 

These  first  stages  of  detection  output  a  list  of  areas  of  attention  in  which  pedestrians  can  be  potentially 
detected.  Each  area  of  attention  is  labelled  using  a  bounding  box.  A  symmetry-based  approach  is  furtherly 
used  to  refine  this  rough  result  in  order  to  resize  bounding  boxes  or  to  separate  bounding  boxes  that  can 
contain  more  pedestrians  [2]. 

These  two  steps  in  the  processing,  barely  take  in  account  specific  features  of  pedestrians;  in  fact,  only 
symmetrical  and  size  considerations  are  used  to  compute  the  list  of  bounding  boxes.  Therefore,  independent 
validators  are  used  to  evaluate  the  presence  of  human  shapes  inside  the  bounding  boxes.  These  stages 
exploits  specific  pedestrian  characteristics  to  discard  false  positives  from  the  list  of  bounding  boxes. 

The  algorithm  described  in  this  report  is  used  as  one  of  the  independent  validators  as  shown  in  figure  1. 
A  final  decision  step  is  used  to  balance  the  votes  of  validators  for  each  bounding  box. 

4  Bounding  boxes  analysis 

In  this  part,  we  will  describe  in  details  the  method  to  classify  the  content  of  bounding  boxes.  As  we  said  in 
previous  section,  the  result  of  stereovision  process  is  a  list  of  bounding  boxes.  Each  bounding  box  produces 
one  image  with  an  object. 

First  we  will  describe  the  image  with  a  descriptor  which  extract  the  discriminant  information  contained 
in  this  image. 

Then  we  analyze  the  descriptor  with  a  classifier,  in  our  case  the  Support  Vector  Machines. 

4.1  HoG  descriptor 

In  2005,  Shashua  et  al.  [22]  had  already  introduced  a  descriptor  using  the  information  of  gradient  orientation. 
But  he  proposed  to  computed  local  histograms  within  small  regions  corresponding  to  the  human  morphology. 
Dalai  et  al.  [11]  has  extended  the  use  of  histograms  but  with  a  dense  approach. 

The  computation  of  a  descriptor  is  done  according  the  following  steps: 

1.  compute  horizontal  Gh  and  vertical  Gy  gradient  of  image  by  filtering  image  with  [—1  0  1] 

2.  compute  both  norm  and  orientation  of  the  gradient: 

•  Ng(x ,y)  =  sj Gh{x,  y )2  +  Gv(x,  y )2  (figure  3-b) 

•  0G{x,  y)  =  atari  (figure  3-c) 

3.  split  image  into  cells  (figure  3-d), 

4.  compute  one  histogram  for  each  cell  (figure  3-e), 

5.  normalize  all  histograms  within  a  block  of  cell. 
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The  last  step  is  a  specificity  of  this  descriptor,  since  the  normalization  can  reduce  the  illumination 
variability.  The  final  descriptor  is  obtained  by  adding  all  normalized  histograms  into  a  single  vector. 
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Figure  3:  Image  characterization  using  HoG:  original  image  (a),  gradient  norm  (b),  gradient  orientation  (c), 
cell  splitting  (d)  and  histogram  computation  (e). 


4.2  SVM  classifier 

The  recognition  system  is  based  on  a  supervised  learning  technique.  Hence,  we  have  used  a  set  of  training 
image  examples  with  and  without  pedestrians,  and  described  by  their  HoG,  to  learn  a  decision  function.  In 
our  case,  we  have  used  a  Support  Vector  Machines  classifier. 

The  Support  Vector  Machines  classifier  is  a  binary  classifier  algorithm  that  looks  for  an  optimal  hyper¬ 
plane  as  a  decision  function  in  a  high-dimensional  space  [4,  30,  8].  Thus,  consider  one  has  a  training  data  set 
{xfc,  yk]  £  X  x  {—1, 1}  where  are  the  training  examples  HOG  feature  vector  and  y the  class  label.  At 
first,  the  method  consists  in  mapping  in  a  high  dimensional  space  owing  to  a  function  <I>.  Then,  it  looks 
for  a  decision  function  of  the  form:  /(x)  =  w  ■  *I>(x)  +  b  and  /(x)  is  optimal  in  the  sense  that  it  maximizes 
the  distance  between  the  nearest  point  4>(x.;)  and  the  hyperplane.  The  class  label  of  x  is  then  obtained  by 
considering  the  sign  of  /(x).  This  optimization  problem  can  be  turned,  in  the  case  of  L\  soft-margin  SVM 
classifier  (misclassified  examples  are  linearly  penalized),  in  this  following  way: 

m 

£ndlwl|2  +  C5>  (!) 

fc=  l 

under  the  constraint  Vfc,  ?/fc/(xfc)  >  1  —  The  solution  of  this  problem  is  obtained  using  the  Lagrangian 
theory  and  it  is  possible  to  show  that  the  vector  w  is  of  the  form: 

m 

w  =  Hafe2/fc$(xfc)  (2) 

k= 1 

where  a*  is  the  solution  of  the  following  quadratic  optimization  problem: 

m  -j  m 

maxIT(a)  =  atkOieykmK('xk,xe)  (3) 

k= 1  k,t 

subject  to  Efcli  Vk^k  =  0  and  V/c,0  <  au  <  C,  where  J£(xfc,x^)  =  (4>(xfc),  $(x^)).  According  to  equation 
(2)  and  (3),  the  solution  of  the  SVM  problem  depends  only  on  the  Gram  matrix  K. 
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5  Results 


In  this  last  part,  we  will  presents  some  results  of  our  system.  We  evaluated  the  HoG  method  with  2  video 
sequences.  The  first:  Tetravision05,  was  taken  during  day  with  good  luminosity  conditions.  The  second: 
TetranightO  1 ,  was  taken  during  night.  For  each  sequence,  we  used  both  a  visible  and  infrared  stereovision 
system. 

Thanks  to  the  tetravision  system  which  was  described  previously,  a  list  of  bounding  boxes  can  be  ex¬ 
tracted  from  all  sequences.  For  each  bounding  box,  we  extracted  the  corresponding  image  and  labeled  it 
manually  as  pedestrian  or  non-pedestrian.  An  image  is  labeled  as  a  pedestrian  if  it  contains  only  one  person, 
which  is  centered.  The  size  of  the  pedestrian  should  also  be  the  size  of  the  bounding  box.  Figure  4  shows 
some  examples  of  pedestrians  and  non-pedestrians.  The  table  below  shows  the  number  of  pedestrians  and 
non-pedestrians  images  which  were  labeled: 


Tetravision05 

TetranightO  1 

FIR 

VIS 

FIR 

VIS 

pedestrian 

2255 

1860 

1678 

1359 

non-pedestrian 

20246 

20520 

2933 

3262 

All  images  are  then  resized  to  the  same  size:  128  x  64  pixels.  This  operation  is  due  to  the  fact  that  the 
descriptor  works  on  images  with  the  same  size. 


Figure  4:  Examples  of  pedestrian  and  non  pedestrian  images  extracted  for  the  tetravision  sequence. 


We  evaluated  independently  each  category:  tetravision  visible,  tetravision  infrared,  tetranight  visible  and 
tetranight  infrared.  We  extracted  randomly  a  subset  of  images  to  build  a  learning  set.  We  learn  the  linear 
Support  Vector  Machine  classifier  on  this  subset  and  test  on  a  random  subset  of  500  images  of  pedestrian 
and  500  non-pedestrian.  It  should  be  notice  that  images  of  the  test  dataset  are  not  images  used  during  the 
learning  step. 

To  improve  the  reliability  of  these  results,  each  test  was  iterated  10  times,  and  we  renewed  randomly  the 
learning  set  and  test  set  for  each  iteration.  The  given  results  are  the  average  of  all  tests. 

To  evaluate  the  performance  of  our  system,  we  compute  the  rate  of  true  positives  against  the  rate  of  false 
positives,  and  we  compute  the  area  under  the  curve  (AUC)  which  is  obtained.  The  table  1  shows  our  results. 
We  also  determined  the  good  recognition  rate  on  the  table  2.  We  also  evaluate  the  performance  when  the 
size  of  the  learning  set  varies. 
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AUC 

10 

50 

100 

500 

Tetranight 

FIR 

0.9554 

0.9602 

0.9662 

0.9704 

VIS 

0.9364 

0.9447 

0.9523 

0.9550 

Tetravision 

FIR 

0.7304 

0.8374 

0.8622 

0.8935 

VIS 

0.7416 

0.8460 

0.8618 

0.8977 

Table  1:  Value  of  Area  Under  Curve  for  each  sequence,  when  the  size  of  the  learning  set  varies. 


Recognition  rate 

10 

50 

100 

500 

Tetranight 

FIR 

0.8727 

0.8993 

0.9060 

0.9144 

VIS 

0.8546 

0.8832 

0.8893 

0.8999 

Tetravision 

FIR 

0.6583 

0.7555 

0.7784 

0.8121 

VIS 

0.6702 

0.7603 

0.7822 

0.8165 

Table  2:  Recognition  rate  for  each  sequence. 


We  can  note  that  we  obtained  very  good  results,  particularly  during  the  night  (Tetranight)  with  up  to 
91%  of  good  recognition  rate  for  the  infrared  night  sequence.  We  can  explain  this  by  the  fact  that  with  the 
infrared  images  the  pedestrian  is  warmer  than  the  background.  Concerning  the  visible  images,  all  pedestrian 
detected  are  pedestrian  located  in  front  of  the  car,  so  images  are  well  defined.  Moreover,  due  to  the  headlight, 
the  pedestrian  could  be  easy  detached  from  the  background. 

During  the  day,  shapes  are  less  contrasted,  so  that  the  characterization  is  not  facilitated.  This  fact  could 
explain  the  lower  performance  achieved  during  day. 

If  we  consider  the  generalization  capacity  of  this  method,  the  performance  is  quite  optimal  for  a  learning 
set  containing  only  50  pedestrian  and  50  non-pedestrian.  When  the  the  learning  set  becomes  larger,  the 
system  performs  better. 

6  Conclusion 

This  report  presented  a  subsystem  for  the  recognition  stage  of  pedestrian  detection  system.  This  module  is 
based  on  use  of  histogram  of  oriented  gradients  combined  with  Support  Vector  Machines  classifier  and  works 
on  both  infrared  and  daylight  images. 

Results  demonstrated  that  this  approach  is  promising;  in  fact,  the  system  is  able  to  correctly  evaluate 
up  to  91%  of  pedestrians  in  the  nocturnal  sequence  and  up  to  80%  in  daylight  scenes. 

Different  ideas  are  currently  under  evaluation  for  improving  the  result.  Particularly,  different  kind  of 
representation  to  characterize  the  image  by  means  of  color  or  texture  can  be  combined.  Also  the  comparison 
of  the  descriptors  can  be  improved  using  various  kernel  formulations,  more  adapted  to  the  oriented  gradient 
descriptor.  Finally,  an  optimization  of  the  computation  time  of  the  recognition  step  in  order  to  assemble 
and  test  under  real  conditions  our  system  is  currently  under  development. 
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